Neural Network Training Methods Research Articles

Моделирование помех в электронном устройстве при воздействии импульсного магнитного поля с использованием искусственной нейронной сети

Modern electronic means are quite sensitive to electromagnetic interference. At the same time, they must function reliably in the existing electromagnetic environment. Lightning discharges and industrial sources make a significant contribution to the formation of the electromagnetic environment around electronic equipment. In this case, pulsed magnetic fields with microsecond parameters most often arise. The most rational approach to ensuring electromagnetic compatibility of electronic means is the most complete accounting and protection from possible phenomena at the design stage. Different methods for modeling the consequences of exposure to electromagnetic interference have their own advantages and disadvantages. To develop a technique and modeling interference in electronic means based on an artificial neural network using the example of the influence of a pulsed magnetic field a the purpose of this work. A practical technique for calculation the magnitude of interference in electronic means using an artificial neural network the paper proposes. All stages of the technique are described: analysis of the main input parameters affecting the amount of interference in the electronic means; the use of a special experimental stand for measuring interference depending on significant input parameters; choosing the structure and parameters of an artificial neural network to modeling interference; choosing a training method for an artificial neural network; choosing a criterion for assessing the quality of training when solving a regression problem; normalization of training data; training an artificial neural network using measured data; modeling the amount of interference in the communication line of an electronic means when exposed to a pulsed magnetic field; assessment of consequences and selection of methods of protection against interference. As an example, we consider the problem of modeling the amount of interference in a communication line inside an electronic means when exposed to a pulsed magnetic field. The magnetic field has parameters recommended by the requirements of the regulatory document on electromagnetic compatibility of devices. In the problem under consideration, an acceptable discrepancy in results is achieved with an acceptable number of neural network training epochs.

The Comparison of the Multi-Layer Artificial Neural Network Training Methods in Terms of the Predictive Quality of the Coefficient of Friction of 1.0338 (DC04) Steel Sheet.

Friction is one of the main phenomena accompanying sheet metal forming methods, affecting the surface quality of products and the formability of the sheet metal. The most basic and cheapest way to reduce friction is to use lubricants, which should ensure the highest lubrication efficiency and at the same time be environmentally friendly. Due to the trend towards sustainable production, vegetable oils have been used in research as an alternative to petroleum-based lubricants. The analysis of friction in sheet metal forming requires an appropriate tribotester simulating the friction conditions in a specific area of the sheet metal being formed. Research has used a special strip drawing tribometer, enabling the determination the value of the coefficient of friction in the blankholder zone in the deep drawing process. Quantitative analysis of the friction phenomenon is necessary at the stage of designing the technological process and selecting technological parameters, including blankholder pressure. This article presents the results of friction testing of 1.0338 (DC04) steel sheets using a strip drawing test. The experimental tests involved pulling a strip of sheet metal between two countersamples with a rounded surface. The tests were carried out on countersamples with different levels of roughness for the range of contact pressures occurring in the blankholder zone in the deep drawing process (1.7-5 MPa). The values of the coefficient of friction determined under dry friction conditions were compared with the results for edible (corn, sunflower and rapeseed) and non-edible (Moringa, Karanja) vegetable lubricants. The tested oils are the most commonly used vegetable-based biolubricants in metal forming operations. Multi-layer artificial neural networks were used to determine the relationship between the value of the contact pressure, the roughness of the countersamples, the oil viscosity and density, and the value of the coefficient of friction. Rapeseed oil provided the best lubrication efficiency during friction testing for all of the tested samples, with an average surface roughness of Sa 0.44-1.34 μm. At the same time, as the roughness of the countersamples increased, a decrease in lubrication efficiency was observed. The lowest root mean squared error value was observed for the MLP-4-8-1 network trained with the quasi-Newton algorithm. Most of the analysed networks with different architectures trained using the various algorithms showed that the kinematic viscosity of the oil was the most important aspect in assessing the friction of the sheets tested. The influence of kinematic viscosity on the value of the coefficient of friction is strongly dependent on the surface roughness of the countersamples.

Controlled descent training

SummaryIn this work, a novel and model‐based artificial neural network (ANN) training method is developed supported by optimal control theory. The method augments training labels in order to robustly guarantee training loss convergence and improve training convergence rate. Dynamic label augmentation is proposed within the framework of gradient descent training where the convergence of training loss is controlled. First, we capture the training behavior with the help of empirical Neural Tangent Kernels (NTK) and borrow tools from systems and control theory to analyze both the local and global training dynamics (e.g., stability, reachability). Second, we propose to dynamically alter the gradient descent training mechanism via fictitious labels as control inputs and an optimal state feedback policy. In this way, we enforce locally optimal and convergent training behavior. The novel algorithm, Controlled Descent Training (CDT), guarantees local convergence. CDT unleashes new potentials in the analysis, interpretation, and design of ANN architectures. The applicability of the method is demonstrated on standard regression and classification problems.

Synthetic data generation methods for training neural networks in the task of segmenting the level of crop nitrogen status on UAV images of agricultural fields

Synthetic data generation methods for training neural networks in the task of segmenting the level of crop nitrogen status on UAV images of agricultural fields

РАСПОЗНАВАНИЕ АКТИВНОСТИ СТУДЕНТОВ В АУДИТОРИИ МЕТОДАМИ ГЛУБОКОГО ОБУЧЕНИЯ

Human activity recognition is one of the actively developing areas of artificial intelligence that can be implemented in learning environments to improve safety, efficiency and quality of education. Deep learning transfer is a widespread method of training neural networks and has proven its effectiveness in such complex tasks as digital image and video processing. In this paper, we propose a model for recognizing different student activities using pretrained models including VGG16, ResNet50, DenseNet201, EfficientNetB0 and Xception, which are then trained using the method proposed in the paper on a Class box image set created by the authors. The EfficientNetB0 model showed the maximum performance among the considered models and achieved an accuracy of 94,25%. The system proposed in this study aims to create a safer and more productive learning environment for students and teachers.

Cost Efficient Training Method for Artificial Neural Networks based on Engine Measurements

Cost Efficient Training Method for Artificial Neural Networks based on Engine Measurements

Real-Time CNN Training and Compression for Neural-Enhanced Adaptive Live Streaming.

We propose a real-time convolutional neural network (CNN) training and compression method for delivering high-quality live video even in a poor network environment. The server delivers a low-resolution video segment along with the corresponding CNN for super resolution (SR), after which the client applies the CNN to the segment in order to recover high-resolution video frames. To generate a trained CNN corresponding to a video segment in real-time, our method rapidly increases the training accuracy by promoting the overfitting property of the CNN while also using curriculum-based training. In addition, assuming that the pretrained CNN is already downloaded on the client side, we transfer only residual values between the updated and pretrained CNN parameters. These values can be quantized with low bits in real time while minimizing the amount of loss, as the distribution range is significantly narrower than that of the updated CNN. Quantitatively, our neural-enhanced adaptive live streaming pipeline (NEALS) achieves higher SR accuracy and a lower CNN compression loss rate within a constrained training time compared to the state-of-the-art CNN training and compression method. NEALS achieves 15 to 48% higher quality of the user experience compared to state-of-the-art neural-enhanced live streaming systems.

Azimuth estimation based on CNN and LSTM for geomagnetic and inertial sensors data

Although estimating the azimuth using a geomagnetic sensor is very useful, the estimation error may be very large due to the surrounding geomagnetic disturbance. We proposed a novel method for preprocessing appropriately for geomagnetic and inertial sensor data to be suitable for the proposed Artificial Neural Network model and training method for the model. As a result, the probability of azimuth estimation error within 1 degree is 96.4% with regression estimation. For classification estimation, when the azimuth estimation probability is 90% or more, the probability that the azimuth estimation error is within 1 degree is 100%.

Multi-source underwater DOA estimation using PSO-BP neural network based on high-order cumulant optimization

Due to the complex and changeable environment under water, the performance of traditional DOA estimation algorithms based on mathematical model, such as MUSIC, ESPRIT, etc., degrades greatly or even some mistakes can be made because of the mismatch between algorithm model and actual environment model. In addition, the neural network has the ability of generalization and mapping, it can consider the noise, transmission channel inconsistency and other factors of the objective environment. Therefore, this paper utilizes Back Propagation (BP) neural network as the basic framework of underwater DOA estimation. Furthermore, in order to improve the performance of DOA estimation of BP neural network, the following three improvements are proposed. (1) Aiming at the problem that the weight and threshold of traditional BP neural network converge slowly and easily fall into the local optimal value in the iterative process, PSO-BP-NN based on optimized particle swarm optimization (PSO) algorithm is proposed. (2) The Higher-order cumulant of the received signal is utilized to establish the training model. (3) A BP neural network training method for arbitrary number of sources is proposed. Finally, the effectiveness of the proposed algorithm is proved by comparing with the state-of-the-art algorithms and MUSIC algorithm.

Prediction of Short-Shot Defects in Injection Molding by Transfer Learning

For a long time, the traditional injection molding industry has faced challenges in improving production efficiency and product quality. With advancements in Computer-Aided Engineering (CAE) technology, many factors that could lead to product defects have been eliminated, reducing the costs associated with trial runs during the manufacturing process. However, despite the progress made in CAE simulation results, there still exists a slight deviation from actual conditions. Therefore, relying solely on CAE simulations cannot entirely prevent product defects, and businesses still need to implement real-time quality checks during the production process. In this study, we developed a Back Propagation Neural Network (BPNN) model to predict the occurrence of short-shots defects in the injection molding process using various process states as inputs. We developed a Back Propagation Neural Network (BPNN) model that takes injection molding process states as input to predict the occurrence of short-shot defects during the injection molding process. Additionally, we investigated the effectiveness of two different transfer learning methods. The first method involved training the neural network model using CAE simulation data for products with length–thickness ratios (LT) of 60 and then applying transfer learning with real process data. The second method trained the neural network model using real process data for products with LT60 and then applied transfer learning with real process data from products with LT100. From the results, we have inferred that transfer learning, as compared to conventional neural network training methods, can prevent overfitting with the same amount of training data. The short-shot prediction models trained using transfer learning achieved accuracies of 90.2% and 94.4% on the validation datasets of products with LT60 and LT100, respectively. Through integration with the injection molding machine, this enables production personnel to determine whether a product will experience a short-shot before the mold is opened, thereby increasing troubleshooting time.

A method of concrete damage detection and localization based on weakly supervised learning

AbstractAutomatic inspection of concrete surface defects based on visual elements is crucial for the timely detection of security risks in infrastructure. Moreover, accurate determination of the geographical location of the detected defects is critical for subsequent maintenance and reinforcement tasks. This study employed convolutional neural network (CNN) training methods for detection and localization. This approach employs bounding boxes to confine damaged pixels and utilizes projection loss to foster similarity learning between pixels. In addition, geotags are automatically indexed through the vectorization of invariant features in a scene, which maintains high model accuracy and reduces training costs. The proposed method can detect and classify typical concrete defects in complex scenarios and accurately locate them without the use of external sensors. In addition, the proposed model can achieve pixel‐level defect detection and geographic location determination through the cost of bounding box annotation and automatic indexing. The proposed model was evaluated using 10,691 images of four typical concrete defects in various complex environments. The results demonstrate that the proposed method achieves a pixel‐level detection accuracy of 48.75 and a location accuracy of 83.69, showing a better performance than other methods.

Models and methods of learning neural networks with differentiated activation functions

Analysis of the literature made it clear that the problem associated with improving the performance and acceleration of ANN learning is quite actual, as ANNs are used every day in more and more industries. The concepts of finding more profitable activation functions have been outlined a lot, but changing their behavior as a result of learning is a fresh look at the problem. The aim of the study is to find new models of optimization tasks for the formulated prob-lem and effective methods for their implementation, which would improve the quality of ANN training, in particular by overcoming the problem of local minima. A studied of models and methods for training neural networks using an extended vector of varying parameters is conducted. The training problem is formulated as a continuous mul-tidimensional unconditional optimization problem. The extended vector of varying parameters implies that it includes some parameters of activation functions in addition to weight coeffi-cients. The introduction of additional varying parameters does not change the architecture of a neural network, but makes it impossible to use the back propagation method. A number of gradient methods have been used to solve optimization problems. Different formulations of optimization problems and methods for their solution have been investigated according to ac-curacy and efficiency criteria. The analysis of the results of numerical experiments allowed us to conclude that it is expedient to expand the vector of varying parameters in the tasks of training ANNs with con-tinuous and differentiated activation functions. Despite the increase in the dimensionality of the optimization problem, the efficiency of the new formulation is higher than the generalized one. According to the authors, this is due to the fact that a significant share of computational costs in the generalized formulation falls on attempts to leave the neighborhood of local min-ima, while increasing the dimensionality of the solution space allows this to be done with much lower costs.

Uncertainty aware neural network from similarity and sensitivity

Recent uncertainty quantification approaches lack transparency. Algorithms often perform poorly in an input domain and the reason for poor performance remains unknown. Therefore, we present a neural network training method that considers similar samples with sensitivity awareness in this paper. In the proposed NN training method for UQ, first, we train a shallow NN for the point prediction. Then, we train another NN to predict absolute differences between targets and predictions. In the next step, we select each sample in the training set one by one and compute both prediction and error sensitivities. Then we select similar samples with sensitivity consideration and save indexes of similar samples. The ranges of an input parameter become narrower when the output is highly sensitive to that parameter. After that, we construct initial uncertainty bounds (UB) by considering the distribution of sensitivity-aware similar samples. Prediction intervals (PIs) from initial uncertainty bounds are larger and cover more samples than required. Therefore, we train bound correction NN. As following all the steps for finding UB for each sample requires a lot of computation and memory access, we train a UB computation NN. The UB computation NN takes an input sample and provides an uncertainty bound. The UB computation NN is the final product of the proposed approach. We have achieved superior performance in most situations through the proposed transparent method. Scripts of the proposed method are available in the following GitHub repository: https://github.com/dipuk0506/UQ

Free association in a neural network.

Free association among words is a fundamental and ubiquitous memory task. Although distributed semantics (DS) models can predict the association between pairs of words, and semantic network (SN) models can describe transition probabilities in free association data, there have been few attempts to apply established cognitive process models of memory search to free association data. Thus, researchers are currently unable to explain the dynamics of free association using memory mechanisms known to be at play in other retrieval tasks, such as free recall from lists. We address this issue using a popular neural network model of free recall, the context maintenance and retrieval (CMR) model, which we fit using stochastic gradient descent on a large data set of free association norms. Special cases of CMR mimic existing DS and SN models of free association, and we find that CMR outperforms these models on out-of-sample free association data. We also show that training CMR on free association data generates improved predictions for free recall from lists, demonstrating the value of free association for the study of many different types of memory phenomena. Overall, our analysis provides a new account of the dynamics of free association, predicts free association with increased accuracy, integrates theories of free association with established models of memory, and shows how large data sets and neural network training methods can be used to model complex cognitive processes that operate over thousands of representations. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

ОЦІНКА ФІНАНСОВОЇ СТІЙКОСТІ ПІДПРИЄМСТВ ЗА ДОПОМОГОЮ НЕЙРОННИХ МЕРЕЖ

The enterprise is the primary link of the economic system, the stability of which is an important condition for the effective development of the national economy. The sustainability of the enterprise covers a set of factors that allow it to achieve a controlled state of equilibrium and the ability for sustainable economic growth through effective management of activities. The main component of the company's stability is its financial condition, which ensures marketing and personnel stability, promotes the development of production and technical-technological stability, maintains investment stability, and increases the efficiency of the management process. The study describes a neural network-based approach to assessing the financial condition of enterprises, which enables the assessment of the enterprise's financial condition based on its annual financial reports with high accuracy (over 90 percent). The study examined different neural network approaches to analyzing financial data, including the use of different neural network types, training methods, and input parameter selection. The article also examines the influence of various financial indicators on the financial state of the enterprise and suggests using the most significant financial indicators as input parameters for neural networks.

Parameter training method for convolutional neural networks based on improved Hausdorff-like derivative

In this paper, we propose a parameter training method in convolutional neural networks (CNNs). To introduce the orders and the cost function into the parameter optimization in CNNs, we give a definition of the Hausdorff-like derivative by analyzing the definition of the Hausdorff derivative, and a corresponding improved form is proposed. The improved Hausdorff-like (IHL) derivative is utilized in the parameter training of the back propagation in CNNs, and the orders are introduced in the back propagation for training of parameters. Therefore, we adjust the orders by judging the size of the cost function during the parameter training in CNNs, and the tuning of the orders can flexibly adjust the training speed of the parameters. An adaptive tuning approach for the orders is presented by analyzing the rule of the orders for the IHL derivative. By combining the proposed parameter training method with the adaptive moment estimation (Adam) algorithm, we propose the Adam algorithm with the IHL derivative. The experiments based on residual networks are carried out, and the experiments show that the proposed algorithm outperforms the original algorithm.

Parameter determination of anisotropic yield function using neural network-based indentation plastometry

Understanding the impact of plastic anisotropy on the formability of sheet metals is crucial for their industrial application and high-precision forming simulation. Existing multiple uniaxial tensile tests for measuring the plastic flow of anisotropic materials, however, are costly, time-consuming, and destructive. Therefore, it is more efficient to focus on indentation plastometry, a simple, nondestructive test that can quickly extract tensile properties. The objective of this work is to directly derive the parameters of the advanced Poly6 yield criterion and hardening, which describes strong plastic anisotropy, using indentation plastometry basd on a neural network (NN) system. The identification process for these parameters through conventional tensile tests is inherently complex, thereby determining the parameters directly from indentation data presents an unprecedented challenge. We trained NNs using a database generated from verified finite element (FE) simulations of spherical indentations. To systematically iterate these FE simulations, we designed a strategy generating a set of input anisotropic parameters that ensure the convexity of the yield function. We considered the radial and vertical displacement fields around the indentation mark along with the load-depth curve as indentation responses. Through a comprehensive analysis of the correlation between displacement profiles, we have proposed an optimal feature extraction method for NN training. The developed FE-NN model was evaluated by comparing the predicted parameters from the indentation responses of the target materials with those measured from tensile tests. These parameters were expressed as the yield locus and directional properties. The results demonstrated that the FE-NN modeling approach is robust and can accurately capture the anisotropic plastic flow from indentation responses.

IDSNN: Towards High-Performance and Low-Latency SNN Training via Initialization and Distillation.

Spiking neural networks (SNNs) are widely recognized for their biomimetic and efficient computing features. They utilize spikes to encode and transmit information. Despite the many advantages of SNNs, they suffer from the problems of low accuracy and large inference latency, which are, respectively, caused by the direct training and conversion from artificial neural network (ANN) training methods. Aiming to address these limitations, we propose a novel training pipeline (called IDSNN) based on parameter initialization and knowledge distillation, using ANN as a parameter source and teacher. IDSNN maximizes the knowledge extracted from ANNs and achieves competitive top-1 accuracy for CIFAR10 (94.22%) and CIFAR100 (75.41%) with low latency. More importantly, it can achieve 14× faster convergence speed than directly training SNNs under limited training resources, which demonstrates its practical value in applications.

A Study on Perception of Visual–Tactile and Color–Texture Features of Footwear Leather for Symmetric Shoes

The study applies Kansei engineering in analyzing the color and texture of leather footwear, utilizing neural network verification to mirror consumers’ visual and tactile imageries onto varieties of leather. This aids in the development of an advanced system for selecting leather footwear based on such impressions. Initially, representative word pairs denoting consumers’ visual and tactile perceptions of leather footwear were delineated. Post-evaluation of these perceptions through a sensibility assessment questionnaire was administered, using 54 samples of leather footwear provided by manufacturers, with each leather type codified in terms of visual and tactile sensibilities. Subsequently, a customized software algorithm was crafted to isolate the primary color and adhesiveness as color features from the leather sample images. Analyzing grayscale values of the images and using pixel neighborhood as a base, the associated calculation methods, such as LBP, SCOV, VAR, SAC, etc., were proposed to extract texture features from the images. The derived color and texture feature values were used as the input layer and the sensory vocabulary quantified values as the output layer. Backpropagation neural network training was conducted on 49 leather samples, with five leather samples used for testing, culminating in the verification of neural network training for three types and 17 combinations. The outcome was an optimal method for leather footwear Kansei engineering and neural network training, establishing a design process for leather footwear characteristics assisted by sensory vocabulary and a backpropagation neural network. Additionally, a computer-aided system for selecting leather footwear, based on these impressions, was designed and validated through footwear design. This study utilized symmetry in footwear design. By using the design of a single shoe to represent the imagery of a pair of symmetrical shoes, we verified whether the leather samples recommended by the leather imagery selection query system met the expected system input settings.

Domain randomization using synthetic electrocardiograms for training neural networks

We present a method for training neural networks with synthetic electrocardiograms that mimic signals produced by a wearable single lead electrocardiogram monitor. We use domain randomization where the synthetic signal properties such as the waveform shape, RR-intervals and noise are varied for every training example. Models trained with synthetic data are compared to their counterparts trained with real data. Detection of r-waves in electrocardiograms recorded during different physical activities and in atrial fibrillation is used to assess the performance. By allowing the randomization of the synthetic signals to increase beyond what is typically observed in the real-world data the performance is on par or superseding the performance of networks trained with real data. Experiments show robust model performance using different seeds and on different unseen test sets that were fully separated from the training phase. The ability of the model to generalize well to hidden test sets without any specific tuning provides a simple and explainable alternative to more complex adversarial domain adaptation methods for model generalization. This method opens up the possibility of extending the use of synthetic data towards domain insensitive cardiac disease classification when disease specific a priori information is used in the electrocardiogram generation. Additionally, the method provides training with free-to-collect data with accurate labels, control of the data distribution eliminating class imbalances that are typically observed in health-related data, and the generated data is inherently private.